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ABSTRACT 



The Accelerated Learning Program (ALP) is the major 
initiative that the Wake County Public School System (WCPSS) is using to help 
all students reach grade level performance in reading and mathematics. In 
2000-2001, ALP's second year of implementation, the program expanded from 
grades 3-8 to grades K-12. The focus of this report is on the program in 
grades 3 through 8. Overall, 10,000 students were eligible for ALP in those 
grades, with 7,325 participating. The program provided up to 22 days of extra 
instruction to students during the school year, with most services provided 
outside the regular school day. The evaluation, which drew on several data 
sources, found changes in both growth and performance that supported the 
effectiveness of ALP and other assistance at grades 3 through 8. The number 
of students able to reach grade level achievement increased over 1999-2000, 
and fewer students dropped from grade-level to below grade-level achievement. 
Both students served in the ALP and those who were not made strong 
achievement gains in most grades in the Wake County schools, and findings 
suggest that extended learning opportunities can make a positive difference, 
but cannot replace other forms of assistance. Some details about the time of 
day ALP services were offered are included. (Contains 26 figures and 2 
references.) (SLD) 
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This paper is based on a larger E&R Report, “ The Accelerated Learning Program (ALP) 2000- 
01: Student Participation and Effectiveness^ ” available at http:/Avww. wcpss. net. 



^SUMMARY 




PROGRAM OVERVIEW 



WCPSS is a large and diverse school system, with over 100,000 students and about 30% minority and 
low income students. We also have urban, suburban, and rural areas within the large county system, 
including 1 1 municipalities. Because of this diversity, our findings may prove useful to other school 
systems as they struggle to meet the educational needs of all students. 



The Accelerated Learning Program (ALP) is the major initiative that the Wake County Public School 
System (WCPSS) is using to help all students reach grade level performance in reading and math. In 
2000-01, ALP’s second year of implementation, the program expanded from grades 3-8 to K-12. The 
focus of this report will be on the program at grades 3-8. 
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Overall, 10,099 students were eligible for ALP in grades 3-8, with most participating (7,325 or 73%). 

The program provided up to 22 days of extra instruction to students during the school year. Most services 
were provided outside of the regular school day, with less than one third of the hours provided during the 
school day. Most of those who chose not to participate in ALP received other forms of assistance, e.g.. 
Special Education, Title I, Language Arts Resource Teachers, English as a Second Language (ESL), 
Project SOAR, Support Our Students (SOS), Communities in Schools (CIS) or private tutoring. 



KEY FINDINGS 



Changes in both growth and performance support the effectiveness of ALP and other assistance at 
grades 3-8: 





• Students in grades 3-8 scoring below grade initially (Level I and II) have shown improved growth 
since ALP began. Level I and II students showed exemplary growth based on the ABC state 
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regression formulas at both the elementary and middle school levels in spring of 2001 . 
(Elementary students initially scoring at Level I or II also showed exemplary progress in 2000.) 
Exemplary growth will be necessary for these students to reach grade level. 

• In 2000-01, the percentage of Level I and II students that were able to reach grade level 
achievement was 45% in reading and 48% in math. This was a considerable improvement over 
1999-00 and previous years in both subjects (when 39% improved in reading and 41% in math). 

• Fewer students also dropped from grade-level to below-grade-level achievement, from about 6% 
to 4.5%. These improved patterns brought a net increase of 1,465 students scoring at grade level 
or above in reading and 850 in math, double the net increase in 1998-99. 

ALP has expanded the system’s capacity for service so that all students at grade level 3-8 scoring below 
grade level can be offered assistance. (In 1998-99, grants and locally funded programs were only able to 
serve 54% of the low achieving students.) 

Both students served in ALP and those students not served made strong gains, exceeding state ABC 
exemplary standards in most grades. At grades 7 and 8, gains for students served through ALP were 
significantly greater than for students served in other ways. Gains were similar for the two groups at most 
other grades, with the exception of grade 3, where students served through ALP gained less than those not 
served. The lower reliability of the third grade pretest may have contributed to these results, along with 
instructional practices such as curriculum pacing, coordination across teachers, lack of sufficient 
grouping, etc. Thus, extended learning opportunities can make a positive difference, but not replace other 
forms of assistance. 

An analysis of system-wide student gains does not suggest any changes in current practices for instructors 
used, group sizes, and hours provided. Findings do suggest that elementary students’ reading gains are 
lower if schools try to provide extra help through ALP both during and outside of the school day. At the 
middle school level, providing ALP during the school day in math seemed to promote the strongest 
achievement growth. 

Comparisons of practices in schools with the highest and lowest gains for low achieving students do 
suggest some optimal practices. The strongest Elementary and middle schools tended to: 

• Use more instructional strategies (especially curriculum mapping), 

• Have steady attendance in ALP throughout the year, and 

• Report fewer problems in recruiting staff than schools with lower growth. 

“High growth” elementary schools had a lower concentration of Level I and II students on their campus, 
and staff mentioned frequent assessment of students as characteristic of their programs. Strong middle 
schools, in comparison with lower growth middle schools, offered ALP at more than one time of day, 
offered ALP throughout the year, teamed across grade levels, and had extended advisories or team time 
more often. 

Extended learning experiences during the school year can make a positive difference, but will require the 
commitment of additional resources and a concerted effort over time. 
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PERSPECTIVES AND FRAMEWORK 



The gap between low and high performing students stands as one of the oldest, most intractable 
problems of education. As standards for student achievement increase, the task of closing the 
gaps becomes even more daunting. To help improve the performance of students who show low 
achievement, states, districts, and individual schools have tried a wide variety of approaches. 
Some efforts have incorporated special instructional programs, while others have focused more 
on overall school reform. State-level approaches currently seek greater accountability by 
defining school progress in terms of absolute performance targets, relative growth, and 
narrowing the achievement gap (Goertz and Duffy, 2001). Myriad approaches have been tried at 
the school level, including formal programs and efforts of schools’ own design (Educational 
Research Service, 1998). 

Cawelti and Protheroe (2001) searched nationwide for school systems that have successfully 
closed achievement gaps between minority and majority students. They found only a few who 
had succeeded, all of which were smaller school systems. The Wake County Public School 
System (WCPSS) is attempting to meet this challenge in a large school system (over 95,000 
students). Improved student achievement is driven by both state accountability standards (which 
include both absolute performance and growth goals for each school) as well as an ambitious 
school system goal, that 95% of students will show grade level performance by 2003 at grades 3 
and 8 (as measured by state tests). The Accelerated Learning Program (ALP) was created to 
provide schools with the necessary resources to give students extended learning opportunities. 

ALP guidelines were established based on research and input from school and central staff. 
Research suggests academic performance is affected by the amount of overall time students for 
instruction, tutoring focused on reading and math, individualized student goals, and frequent 
assessment of student progress (American School Board Journal, 1998; Accelerated Learning 
Program, 1 999). Parameters specified that students who scored below grade level were to be 
provided up to 22 days of additional instruction (occurring outside of the regular school day) 
through trained instructors (preferably teachers) in small groups (of 15 students or less). Within 
these parameters, schools were given latitude in their implementation approaches. Schools were 
allotted funds for each student in grades 3-8 scoring in Level I or II (considered below grade 
level on EOG). Schools with the highest percentage of low-income students received extra 
funds. The ALP Program was funded through local and state funds. 

Assistance to students continued to be available through programs such as Special Education, 
Title I, Language Arts Resource Teachers, English as a Second Language, and Communities in 
Schools. 

OBJECTIVES 

In this paper, we will describe: 

• How extended day opportunities were provided to students. 

• The achievement effects of providing extended learning opportunities to students who 
initially show below grade level achievement. 
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• Program features at the school and system level that led to greater success for students. 

METHODS 

The emphasis of this evaluation was on ALP. However, the Evaluation and Research 
Department (E&R) also collected and analyzed data on all assistance programs available to 
students considered to be below grade level — those who scored below grade level on the End-of- 
Grade (EOG) state assessments or the district’s literacy and math assessments at the end of grade 
2. This allowed us to study the potential differential impact of various services for students, as 
well as the potential overlap in services. The basic questions guiding our evaluation efforts were: 

• What services were available to students considered below grade level? 

• How was ALP designed and implemented? 

• How many students participated in ALP and other programs? 

• What was the impact of ALP and other services on the achievement of students who were 
low achieving in general? 

• What was the impact of ALP compared to other services for these students? 

• What program-related factors led to the greatest gains for these students system-wide? 

• What school-related factors led to the greatest gains for these students? 

We utilized both quantitative and qualitative analyses in this study. Descriptive analyses were 
used extensively to summarize school and system overall approaches to implementation, and 
staff impressions of effectiveness. Analyses of state test results included both descriptive 
statistics on scale score and level score changes over two years as well as regression analyses of 
two types. State regression formulas were used to assess schools’ and subgroups effectiveness 
relative to state expectations for expected and exemplary growth. In addition, we carried out 
additional regression analyses to explore whether various program facets impacted student 
achievement results. The spring 2000 EOG scores were regressed on the hours of help provided, 
instructional group size, type of instructor, timing of help, and instructional approaches. The 
spring 1 999 EOG scores were used as a control variable. After deleting missing data across all 
variables, 6,195 students were in the reading analysis and 5,452 in the math analysis. 

DATA SOURCES 

The primary data sources for this evaluation included the following. 

• Surveys to collect ALP program descriptions from each school and feedback on its 
effectiveness for them, 

• Data sheets for each of approximately 1 0,000 students eligible for ALP, detailing services 
provided, when, and by whom, 

• Scale scores and level scores on the state’s End of Grade (EOG) multiple-choice tests in 
reading and math for spring 1 999 and 2000 (pre and post), and 

• District central computer files listing important demographic information on students such 
as days of enrollment and other program services received within WCPSS. 
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ALP PROGRAM 



Implementation: All schools implemented ALP. Nearly all students scoring in Levels I and II 
(about 90%) received additional assistance from ALP or another program. 

• Most traditional-calendar schools offered ALP on Saturdays and after school. Year-round 
schools served students during intersessions primarily, with some after-school and Saturday 
sessions. 

• Schools offered an average of 100 hours for ALP students. The number of hours offered per 
school varied widely, from 35-505 hours. 

• Overall ALP attendance was higher for elementary (72%) than middle schools (63%). At 
traditional-calendar schools, attendance rates were higher for sessions held on school days 
than for sessions on non-school days. Intersession attendance was high at year-round 
schools. 

• All schools used teachers from their own school to deliver ALP instruction. In addition, 
schools utilized volunteers, teacher assistants, other school professionals, retired teachers, 
and teachers from other schools. 

ALP PARTICIPATION 3-8 

As the following figure illustrates, 10,984 students initially were eligible as of spring 2000 
(compared to 10,464 in spring 1999). More students were technically eligible compared to 
spring 1999 because eligibility rules were adjusted to include all special education students not 
tested on the EOG. This was done in response to a change in state rules, which required testing 
with some form of the EOG for all special education students (standard test, computer adaptive 
test, checklist, or portfolio). Students assessed with some alternate form of the EOG test were 
less likely to be instructed on the standard course of study at grade level, and therefore less likely 
to want to participate in ALP. However, there was no way to predict which students to exclude, 
so all were therefore counted in eligibility counts. 

Some students (231) were added to the original eligibility count because they were new to the 
system or in need based on school assessments. (Schools were allowed to serve students that did 
not score below grade level on the EOG if there was room in their program once eligible students 
were invited to attend.) On the other hand, 621 students left WCPSS during the school year. As 
of spring, 2001, 10,099 students were enrolled and eligible to participate in the Accelerated 
Learning Program. Of these, 7,325 (72.5%) participated in ALP; 2,744 (27.5%) did not. Most 
of those who opted not to participate in ALP were served through another WCPSS program 
(only 10%, or about 1,000 students, were not). About 45% of those eligible were actually served 
through ALP and another form of assistance. 
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The overall allocation for the ALP program at grades 3-8, including the challenged schools, 
school grant, and community components, was $6,805,21 1. With service to 7,325 students, the 
cost per student was $929. 



Figure 1 

ALP 3-8 Eligibility and Participation 2000-01 



Spring 00 




Criteria: 


Eligible for ALP 00-01: 




• LI -II scores or Special 


10,984 Students 




Education (grades 4-8) 






• Low literacy or math profile 




(grade 3) 



i 



Fall 00 




Eligible pool reduced by 
723 students to 10,261 




Reduced because: 

• 717 eligible leave WCPSS 

• 6 retained in grade 2 





Spring 01 

• Eligible pool reduced by 
392 students to 9,869 
students 



. Reduced because: 

392 eligible leave WCPSS 



1 




Eligible pool 
increased by 

• 230 students (new to 
WCPSS or served 
without LI-II EOG) 

• 1 0,099 total students 
eligible 



i I 



ALP 




ALP Non- 


Participants = 




Participants = 


7,325 students 




2,774 students 


(72.5%) 




(27.5%) 



Service for non-participants: 

• 1,627% (16%) in Special 
Education, mostly in “separate 
settings” 

• 70 (1%) in ESL 

• 63 ( 1 %) in Title I, SOS, SOAR 

• 1,039 (10%) in no WCPSS 
program 
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Compared to WCPSS overall, Black and low-income students were over-represented in those 
eligible for the program, a trend that has been found for assistance programs over time. Of those 
eligible based on test performance: 

• 58% were Black, with 46% of other ethnic backgrounds, 

• 2.3% were in English as a Second Language (ESL) programs, 

• 54% were male and 46% were female, 

• 43% received special education services, and 

• 50% were low income (based on free- or reduced-price lunch eligibility). 



Figure 2 

ALP 3-8 Percentage of Students Eligible and Participating, 
with Comparative Characteristics to WCPSS Overall 



Ethnicity 


ALP 3-5 


ALP 6-8 


ALP 3-8 


%of 

K-12 

Population 


Eligible 


Participating 


Eligible 


Participating 


Eligible 


Participating 


White 


33.1 


30.8 


31.1 


29.5 


32.2 


30.5 


• 63.2 


Black 


55.7 


58.0 


60.0 


61.6 


57.6 


57.5 


26.2 


Hispanic 


7.5 


7.5 


5.9 


5.4 


6.8 


6.7 


4.6 


Asian 


1.6 


1.4 


1.6 


1.8 


1.6 


1.6 


3.9 


Native 

American 


0.2 


0.3 


0.2 


0.1 


0.2 


0.2 


.2 


Multi/Other 


2.0 


2.0 


1.1 


1.1 


1.6 


1.6 


1.8 




ESL | 3.0 2.7 | 1.5 1.5 | 2.3 2.2 | 3.4 




Male 


54.2 


53.0 


56.9 


55.9 


55.4 


54.2 


51.0 


Female 


45.8 


46.0 


43.1 


44.1 


44.6 


45.8 


49.0 ‘ 




FRL 


51.8 


52.0 


50.2 


47.6 


51.0 


50.2 


20.0 


Non-FRL 


48.2 


48.0 


49.9 


52.5 


49.0 


49.8 


80.0 




Special 

Education 


41.6 


36.0 


45.4 


39.3 


43.3 


37.3 


13.5 




Total 

Counts 


5,634 


4,342 


4,465 


2,983 


10,099 


7,325 


97,728 



Note: Participation as of spring 2001 . Percentage of population is from 20th-day counts, fall 2000. 



Participation rates mirrored eligibility rates closely for ethnicity, ESL, gender, and income. For 
example, about half of the students eligible and participating in ALP were low income and about 
half were not. Special education participation (37% of those served) was slightly lower than the 
percentage eligible for service (43%). On closer examination, the lower participation rate was 
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for students receiving special education in separate settings. These students also generally have 
the greatest special learning needs. 



Figure 3 

Percentage of Special Education Students Participating in ALP 



90% -| 

80% - 
70% - 
60% - 
50% - 
40% - 
30% - 
20 % - 
10% - 
0 % - 



84.6% 85.2% 84.8% 




74.5% 



72 - 4% 70.9% 



□ Not Special Ed 

□ Regular Special Ed 
B Resource Special Ed 

I Separate Special Ed 




ALP 3-5 



ALP 6-8 



The figure below shows that participation rates were generally higher at elementary than middle 
schools, with the highest participation at grade 4 and the lowest at grade 8. 

Figure 4 

ALP 3*8 Eligibility and Participation by Grade 



GRADE 


# Eligible 


# Participating 


% Participating 


3 


2,062 


1,538 


74.6 


4 


1,894 


1,521 


80.3 


5 


1,678 


1,283 


76.5 


3-5 


5,634 


4,342 


77.1% 


6 


1,515 


1,036 


68.4 


7 


1,732 


1,173 


67.7 


8 


1,218 


774 


63.6 


6-8 


4,465 


2,983 


66.8% 


TOTAL 3-8 


10,099 


7,325 


72.5% 



Of the 7,325 participants, most participated in programs in both reading (6,634 or 91%) and math 
(6,006 or 82%). (Some students did not score low in both subjects.) 
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ALP 3-8 ACHIEVEMENT TRENDS FOR LEVEL I AND II STUDENTS 



Since ALP was such a large intervention effort and participation rates were high, examining 
student growth for all students who scored in Levels I or II in spring of 2000 provides useful 
information on overall program impact. 



Overall, results showed quite positive achievement trends: 

• The percentage of students moving up to grade level on EOG (from Level I-II to III- 
IV) increased. 

• The percentage of students moving down from Level III-IV to Level I-II decreased 
(from 6% to 4.5%). 

• The net increase in students in Level III-IV (1 ,465 in reading and 850 in math) was 
double that of 1999-2000. 



Percentage of Students Able to Move Up to Grade Level 2000-01 

WCPSS showed improvement in the percentage of students able to improve their performance 
from below grade level on EOG (Levels I-II) to at or above grade level (Levels III-IV) than was 
true in past years. 

Figure 5 

Percentage of Students Who Moved Up to Grade Level 



100% n 



80 % - 



m 1997-98 

□ 1998-99 
■ 1999-00 

□ 2000-01 



60 % - 




Reading Math 



Reading 



10 , 



3 
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More students were able to move up from Level I-II to Level III- IV between spring of 2000 and 
2001 than with previous cohorts. Of those who scored Level I or Level II in spring 2000, 45% 
moved up to Level III or Level IV in reading, compared to 39% the previous year. 

• The greatest improvements were seen in grades 3 and 5 in reading. 

• Grade 6 in reading stands out as an area for improvement in moving students up across 
the year. Only 22% of students scoring below grade level in spring of 2000 were able to 
move to on grade level scores in spring of 2001, compared to 41-57% in the other grades. 

The increase seen in 2000-01 represents a jump from previous years, and likely reflects the more 
extensive program provided in 2000-01, schools’ greater experience with the program, and the 
cumulative impact of more than one year of help for many students. 

A check of improvement among special education students revealed that 34% of those served 
were able to improve from Level I-II between spring of 2000 in reading and 38% in math. 
Although these percentages are about 10% lower in each subject than the overall rates, they do 
represent substantial improvement for a group with special learning needs. 

Figure 6 

WCPSS Achievement Level Status on EOG Reading between Spring 2000 and 2001 



EOG READING 


Grade 


Spring 2001 Status (Posttest) 


Level I-II 

Spring 2000 (Pretest) 


2000-01 


LI-D# 


LI-D % 


LIII-IV # 


Lm-iv % 


3 


683 


48.79 


717 


51.21 


4 


613 


5833 


438 


41.67 


5 


499 


43.09 


659 


56.91 


6 


755 


78.16 


211 


21.84 


7 


774 


57.12 


581 


42.88 


8 


419 


48.95 


437 


51.05 


All 


3,743 


55.16 


3,043 


44.84 


Level III-IV 
Spring 2000 (Pretest) 


2000-01 


LI-D # 


LI-D % 


LIII-IV # 


LIII-IV % 


3 


357 


6.02 


5,572 


93.98 


4 


305 


5.15 


5,623 


94.85 


5 


108 


1.86 


5,701 


98.14 


6 


506 


8.43 


5,499 


91.57 


7 


179 


3.26 


5309 


96.74 


8 


123 


2.17 


5356 


97.71 


All 


1,578 


4.53 


33360 


95.47 


Net Increase in Students in Levels 

m-iv 


2000 to 2001: 
3,043-1,578= 1,465 


1999 to 2000: 
2,646-1,972= 674 



Fewer students also dropped from Levels III or IV to Levels I or II between spring 2000 and 
spring 2001 than with previous cohorts. Of those scoring on grade level in spring 2000 in 
reading, 4.5% moved down to below-grade-level scores in spring 2001, compared to 6% the 
previous year. 
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• The smallest declines were evident in grades 5 and 8 (about 2%). 

• The greatest decline was evident at grade 6 (over 8%). 

The more positive patterns in results led to a net increase of 1,465 students showing grade-level 
achievement in reading — more than double the 67- student net improvement obtained last year. 

Math 

As in reading, more students were able to move up from Level 1-11 to Level 111-TV between spring 
of 2000 and 2001 than with previous cohort. Of those who scored Level I or II in spring 2000, 
48% moved up to Level III or IV in reading, compared to 41% the previous year. 



• The greatest improvement was seen in grade 4. 

• The least improvement was evident in grades 3, 7, and 8. At grade 3, the reliability of the 
pretest is lower than at other grades. However, greater relative difficulty in improving 
students’ scores may also relate to the fact that additional assistance is not provided in 
math until grade 3 (unlike reading, which starts in the earlier grades). 

Figure 7 

WCPSS Achievement Level Status on EOG Math: Spring 2000 and 2001 



EOG Math 


Grade 


Spring 2001 Status (Posttest) 




2000-01 


LI-II # 


LI-II % 


liii-iv # 


LIII-IV % 




3 


581 


66.55 


292 


33.45 


Level I-II 

Spring 2000 (Pretest) 


4 


416 


32.58 


861 


67.42 


5 


337 


51.61 


316 


48.39 


6 


468 


55.91 


369 


44.09 




7 


505 


58.25 


362 


41.75 




8 


428 


58.55 


303 


41.45 




All 


2,735 


52.21 


2,503 


47.79 




2000-01 


LI-II # 


LI-D % 


LIII-IV # 


LIII-IV % 




3 


539 


8.32 


5,937 


91.68 




4 


52 


0.91 


5,687 


99.09 


Level III-IV 


5 


184 


2.90 


6,171 


97.10 


Spring 2000 (Pretest) 


6 


285 


4.63 


5,866 


95.37 




7 


255 


4.27 


5,721 


95.73 




8 


338 


5.83 


5,462 


94.17 




All 


1,653 


4.53 


34344 


95.47 


Net Increase in Students in Levels 


2000 to 2001: 


1999 to 2000: 


III-IV 




2,503-1,653= 850 


2,363-1,940=423 



Fewer students also dropped from Levels III or TV to Levels I or II between spring 2000 and 
spring 2001 than with previous cohorts. Of those scoring on grade level in spring 2000 in 
reading, 4.5% moved down to below grade level scores in spring 2001 , compared to 6% the 
previous year. 
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• The smallest declines were evident in grades 4 and 5. 

• The greatest decline was evident at grade 3. 

The more positive patterns in results led to a net increase of 850 students showing grade level 
achievement in math, double the net improvement of 423 students obtained the previous year. 

Student Score Declines Spring 2000 to 2001 

In spring of 2000, the vast majority of WCPSS scored in Levels III or IV; most scored well 
above the score that divides students into Level II or III (“cut point for below and above grade 
level)”. The standard error on the EOG at most grade levels is two points. (The percentage of 
students who scored five or more points above the Level II-III cut point was 84% in reading and 
90% in math.) The next chart illustrates that: 

• Most students were able to score at Levels III or IV again in 2001 regardless of their 
initial closeness to the cut points. 

• Those closest to the cut were most likely to decline to Level I or II. 

• Of those who declined, over half (62% in reading and 59% in math) initially scored 
within four points of the scale score cuts. 



Figure 8 

Level III-IV Students from Spring 2000 Declining to Level I or II as of Spring 2001 
in Relation to Level II-III Scale Score Cut Points 



Points Above Level II-III Cut 


Spring 2000 
Status 


Spring 2001 Declines to Level I-II 


Reading 


# Students 


# 


% 


Level III 1 -2 points above cut 


2,086 


536 


25.7% 


Level III 3-4 points above cut 


2,490 


443 


17.8% 


Level III or IV 5 or more points above cut 


30,404 


599 


1.9% 


Total 


34,980 


1,578 


4.5% 


Math 








Level III 1-2 points above cut 


1,569 


487 


31.04% 


Level III 3-4 points above cut 


2,027 


489 


24.12% 


Level III or IV 5 or more points above cut 


33,069 


677 


2.05% 


Total 


36,665 


1,653 


4.5% 



Overall Results Relative to ABC Standards 



The state’s ABC regression formulas provide one yardstick by which we can assess the adequacy 
of growth for our students from one year to the next. Expected growth represents scale-score 
growth expected of students over one year; exemplary growth is approximately 1 10% of 
expected growth. Overall, the WCPSS school system showed exemplary growth at the 
elementary level and expected growth at the middle school level. 
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ABC results can also be broken down by initial achievement level. Strong exemplary growth is 
needed with our Level I and II students if we hope to achieve the 95% achievement goal over 
time. Although all students who score in Levels I or II do not move up to Levels III or IV 
performance in one year, consistently strong scale-score growth over time should lead most 
students to grade-level performance. 

The following figure shows the actual scale-score growth shown for WCPSS students who 
scored in Levels I or II between the spring 2000 and spring 2001 , along with the expectations for 
these students based on the state’s ABCs of Accountability regression formulas. Expected and 
Exemplary Growth of 0 or above means the standard for each was met. 

• At the elementary level, WCPSS showed exemplary growth for students who initially 
scored in Levels I or II and Level III for the second year in a row. WCPSS showed 
expected growth for those initially scoring in Level IV. This is a positive pattern, in that 
all students are growing, but those with the greatest need for improvement are growing 
the most. 

• At the middle school level, WCPSS showed exemplary growth for students who initially 
scored in Levels I or II and expected growth for those initially scoring in Levels III or IV. 
This is a more positive pattern than was found last year and again reflects strongest 
growth for those with the greatest need for improvement (Level I and II students). 

• At both the elementary and middle school grades, growth was stronger for students in 
Levels I and II than for the students receiving free or reduced-price lunches and Black 
students. Our students in Level I or II who are low income and/or Black represent the 
greatest challenge for WCPSS educators. 

Figure 9 

ABC Exemplary Growth for Key Subgroups in 2001 



Group 


Grades 3-5 


Grades 6-8 . 


Systemwide 


.10 


-.04 


Levels I and II 


.77 


.24 


Level III 


.10 


-.19 


Level IV 


-.22 


-.03 


Low Income 


-.20 


-.66 


Not Low Income 


.19 


.08 


Black Males 


-.26 


-.57 


Black Females 


-.16 


-.48 


White Males 


.20 


.05 


White Females 


.21 


.16 



Bold indicates ABC Exemplary Growth Standard was met. 



signifies best results by level. 
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Breaking down the results for students who initially scored at Levels I or II by grade shows that: 

• The strongest growth in reading was evident at grades 5, 7, and 8. Grades 3 and 6 
showed the greatest need for improvement relative to ABC exemplary standards. 



• The strongest growth in math was evident at grades 4, 5, and 7, with grades 3, 6, and 8 
showing need for improvement to reach exemplary growth standards. 

Thus, grades 3 and 6 showed the most need for improvement. A Sixth-Grade Task Force has 
been formed to explore possible reasons and solutions for grade 6 (lower gains reflect a state- 
wide trend in recent years). At grade 3, possible reasons could be related to instruction or 
assessment issues. The fall test at 3rd grade is much shorter and less reliable than at the other 
grades. 



ALP 3-5 PROGRAMS 

STUDENT ELIGIBILITY AND PARTICIPATION 

Data sheets returned for individual students in grades 3-5 indicated that 77. 1% of the eligible 
students actually participated in the program. Most of those who did not participate were served 
in other ways. The next figure summarizes enrollment breakdown by grade. 



Figure 10 

Enrollment in ALP 3-5, 2000-01 



Grade 


# Students Eligible 
for ALP 


# Students Who 
Participated in 
ALP 


% Students Who 
Participated in 
ALP 


3 


2,060 


1,537 


74.6% 


4 


1,896 


1,522 


80.3% 


5 


1,677 


1,283 


76.5% 


Total 


5,633 


4,342 


77.1% 



Source: Instructional Assistance Data Sheets 



Participation rates varied by school from 39.2% to 100% of those eligible. Other services 
available to students in grades 3-5 included: 

• Title I 

• Special Education 

• English as a Second Language 

• Communities in Schools 

• Project SOAR 

• Parent and other volunteer tutors 

• Small group and individual instruction provided by special area teachers, the LART, the 
IRT, the lead literacy teacher, and/or teacher assistants. 
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ATTENDANCE 



Overall attendance rates for ALP varied depending upon the days and times the program was 
offered and the school level (elementary and middle school). 

• For elementary schools, sessions offered during the day were most heavily attended (95% 
mean attendance), followed closely by after-school sessions (90% mean attendance). 

• Sessions on intersession days at year-round schools reported attendance of 84%. 

• At traditional schools, sessions held on days when school was not in session had lower 
attendance (60-63%). 



Figure 11 

2000-01 Attendance Trends for ALP 3-5 



Student Attendance 


Saturday 
(N = 41 of 76 
schools) 

% of schools 
reporting this 
attendance rate 


Before/ 
After School 
(N = 46 of 76 
schools) 

% of schools 
reporting this 
attendance rate 


During the 
Day 

(N = 19 of 76 
schools) 

% of schools 
reporting this 
attendance rate 


Intersession 
(N = 9 of 9 
schools) 

% of schools 
reporting this 
attendance rate 


Under 60% 


31.7% 


10.9% 


10.5% 


33.3% 


60-69% 


31.7% 


2.2% 


0 


0 


70-79% 


14.6% 


6.5% 


5.3% 


11.1% 


80-89% 


12.2% 


17.4% 


5.3% 


22.2% 


90% and Above 


9.8% 


58.7% 


78.9% 


33.3% 



Source: ALP Feedback Forms (76 of 78 schools reporting) 



Figure 12 

ALP 3-5 Attendance Across Schools: Mean Percentages 





On School Days 


Not on School Days 


During the 
Day 


Before 

School 


After 

School 


Teacher 

Workday 


Saturday 


Inter- 

session 

Days 


2000-01 


95% 


80% 


90% 


60% 


63% 


84% 


1999-2000 


— 


82% 


85% 


56% 


63% 


88% 



Source: ALP Feedback Forms 
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Individual students’ attendance averaged about 70% based on our sample of 1,555 students. 

More information on hours of service can be found in the Student Achievement Outcomes 
section of this report. 

SUCCESSES AND CHALLENGES 

Based on the ALP Feedback Forms, elementary schools considered the greatest success of ALP 
to be staff commitment (65.4%), followed by student enthusiasm and students’ learning (56.4% 
each). Schools found that recruiting staff (41%) and staffbumout (35%) were the most 
challenging aspects of their ALP program. 

Russonello and Stewart (2001) recently surveyed pre-K-8 principals nationwide about after- 
school efforts. The biggest challenges principals noted were obtaining sufficient funds, finding 
and retaining good staff members, and securing transportation for students. Other than funding, 
WCPSS school staff mentioned similar issues. (Our school staff may have considered funding to 
be a system challenge rather than one for their school.) 

STUDENT ACHIEVEMENT OUTCOMES 
Growth for Students Based on ALP Participation 

As presented earlier, Level I and II students made strong growth between spring 2000 and spring 
2001 . The issue addressed here is whether students who participated in ALP showed stronger 
growth than those who did not. We analyzed this question in two ways. First, we compared 
growth for students served in ALP (or ALP and another program) to those not served in ALP 
(most of whom were likely served in another way, e g., exclusively through Title I, Special 
Education, ESL, SOAR, SOS, or a private tutor). Analyses were subject specific, with only 
students who scored in Levels I or II in reading as of spring 2000 (fall 2000 for grade 3) included 
in the reading analyses, and only those who scored low in math initially included in the math 
analyses. We utilized ABC expectations for exemplary growth for Level I and II students as an 
external standard for effectiveness. The ABC state regression model was designed to show the 
kind of progress students who are low achievers need to demonstrate in order to reach grade 
level between grades 3 and 8). ABC analyses include students with Level I or II in either 
reading or math in analysis of both subjects. Our analysis was more accurate in tracking 
improvement for those who truly had deficits, but results must be interpreted with more caution 
because of this difference in inclusion rules. 

We then checked the characteristics of eligible students who chose to participate in ALP and 
those who did not choose to participate. Compared to students not in ALP, those in ALP were 
about as likely to be low income (52% of those in ALP were low income, compared to 51% of 
those not in ALP), but much less likely to be participating in special education programs (36% 
vs. 60%). We therefore conducted a second set of analyses using regression, and controlled for 
pretest scores, ffee-lunch status, and special education status. One caution is that the two groups 
may have varied in other ways as well. Based on the student sample for whom we collected 
more information, another service was considered more appropriate to student needs in many 
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cases (mostly special education, though a few mentioned services such as ESL or tutors). Late 
school enrollment, lack of test scores, and scheduling conflicts were mentioned in a few other 
cases. 

As the following figure shows, mean scale-score gains for elementary students in ALP and/or 
other programs met ABC exemplary standards in all grades in math and grades 4 and 5 in 
reading (missing the benchmark only at grade 3). These overall results are positive and point out 
the benefit of making assistance available to all those in need (which was not possible before 
ALP). 

On the other hand, ALP did not appear to result in stronger gains than other support available to 
students at the elementary level. Student growth appeared similar for those in ALP and those not 
in ALP at grades 4 and 5 and significantly lower for third graders in ALP than not in ALP. 
Similar gains for the two groups are not a concern when both groups showed strong growth. 
Similarly strong growth suggests that students are receiving appropriate support. However, 
third-grade results deserve some follow-up. The shorter, less reliable pretest at grade 3 
contributes to results that must be interpreted cautiously. However, it is also worth considering 
ways to improve the overall instructional program and ALP specifically at grade 3, especially 
since it is a grade targeted in the system’s achievement goal. 
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Figure 13 

EOG Scale Score Gains Based on ALP 3-5 Participation 
Relative to ABC Exemplary Standards for Level I-II Students in 2000-01 
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Reading Math 



Grade 3 




Reading Math 



Grade 4 



■ ABC Exemplary 
0ALP 
□ Not ALP 




Reading Math 



Grade 5 



Figure 14 

ALP 3-5 EOG Reading and Math Gains for Students in Level I-II in ALP or Not Served in 
ALP (Most of Whom Were Served Through Another WCPSS Program) 



EOG Reading 


Grade 3 


Grade 4 


Grade 5 


# Students 


Gain in SS* 


# Students 


Gain in SS* 


# Students 


Gain in SS* 


ABC Exemplary 


1,350 


11.3 


1,503 


5.9 


1,248 


7.1 


ALP 


711 


9.8 


856 


7.5 


958 


10.5 


Not ALP 


119 


10.6 


128 


7.8 


136 


10.3 


All Level I-II 


830 


9.9 


984 


7.6 


1,094 


10.5 


EOG Math 


Grade 3 


Grade 4 


Grade 5 


# Students 


Gain in SS* 


# Students 


Gain in SS* 


# Students 


Gain in SS* 


ABC Exemplary 


1,350 


15.8 


1,503 


9.3 


1,248 


7.5 


ALP 


516 


17.6 


991 


14.3 


506 


12.2 


Not ALP 


85 


17.0 


199 


14.5 


101 


10.8 


All Level I-II 


601 


17.5 


1,190 


14.3 


607 


12.0 



*Gains in SS = Average Scale Score Gain 00 to 01 

Note 1: Shading Means Group Met ABC Exemplary Growth Standard 

Most students not served through ALP were served in other WCPSS programs. 
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FACTORS LEADING TO GREATEST ACHIEVEMENT GAINS 



Schools had a great deal of flexibility in implementing ALP (within broad guidelines), and 
student gains varied considerably across school programs. We therefore examined whether 
student gains differed based on program characteristics. One regression analysis analyzed the 
impact of the time of service on student achievement outcomes, while a second explored the 
impact of other program characteristics. Specifically, we addressed two questions: 

• Did the time of day of instruction (during, outside the school day, or both) impact student 
scale score achievement gains? 

• Did factors related to other program characteristics (type of instructor, hours of service, and 
group size) impact the size of the achievement gains made by students? 

Across grades 3-8, the analyses were based on the sample of 1,855 students, of which 1,551 
responded (84%). Overall, 945 students were included in the reading analysis and 843 in the 
math analysis. To be eligible for the analysis, students had to have pre- and posttest scores and a 
Level I or II score in the subject in spring of 2000 and 2001 . Students also had to have data on 
all variables in the analysis. Due to small sample sizes at some grades, analyses were done by 
level (grades 3-5 and grades 6-8). Because expectations for scale score gains and sample sizes 
vary by grade on the EOG, we decided to use residual scores from the effectiveness index for 
these analyses. These residual scale scores reflected the difference between students’ actual 
gains between spring of 2000 and 2001 and what was expected of them (based on a regression 
formula which controlled for special education and free lunch status). The residual scores were 
first standardized for analysis through conversion to z scores. 

Timing of ALP Service 



Schools were given more flexibility in 2000-01 to provide ALP services at least partially during 
the day. Most students continued to receive ALP help primarily outside of the regular school 
day in 2000-01 (about 70%), with 15% served both during the day and outside for ALP another 
1 5% served only during the day. The regression model tested whether students were more likely 
to show higher standardized residual scores if they were served only outside of the school day, 
during the school day, or both during and outside of the school day. Overall for the elementary 
analysis, 632 students were included for reading and 583 for math. As the next chart illustrates, 
time of service did not impact elementary math gains, but reading residuals were lower if 
students were served both during and outside of the school day (compared to outside only). 
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Figure 15 

Importance of Help Outside the School Day: Elementary 





Time of Day 


N 


Significance 


Reading 


Outside Only 


442 


Significantly higher than Both (p=.027) 


During Only 


88 


Ns 


Both 


103 


Significantly lower than Outside Only (p=.027) 


Math 


Outside Only 


483 


ns 


During Only 


73 


ns 


Both 


72 


ns 



These results suggest that math ALP programs can be successful regardless of the time of day 
students are served. However, with elementary students, it is better not to serve students in 
reading through ALP both during and outside of the school day. This may reflect schedule 
and/or planning coordination issues or differences in instructional approaches across teachers. 

Other Factors Related to ALP 3-5 Service: Attendance. Group Size, & 
Instructor 

In addition to examining the timing of ALP, the analysis also considered attendance, group size, 
and instructor. After deleting missing data across variables within subject, 551 elementary 
school students were included in the reading analysis, 498 in the math analysis, and 235 in the 
writing analysis. Descriptive statistics for attendance, group size, and instructor are shown in the 
following figure. 

On average, elementary students planned and attended more ALP sessions in reading (36) than in 
math (30) and writing (28). Rates of attendance (79%) and hours per session (1.6) were 
comparable in reading and math but slightly lower in writing. Total service, as derived by 
multiplying the number of sessions attended by the hours per session, was higher in reading than 
in math, and higher in math than in writing. (Less service in ALP in writing is reasonable, since 
it is tested less often by the state and is not part of the system’s achievement goal.) 

Group size averaged around 9 students in both reading (8.8) and math (9.1) and writing (9.6). 
Group size had a standard deviation of about 2.7 students for reading and math and 4.0 for 
writing. 

The instructor variable consisted of three categories for ALP, reflecting service by: a) the 
student’s teacher (often plus others), b) another teacher only, or c) other professionals, teaching 
assistants, and volunteers. As shown in the next figure, students were least often taught by their 
own teacher and most commonly taught by teachers other than their own. 

■ For both reading and math, only about 12% received instruction from their own teachers. 
About 60% of students were taught only by teachers other than their own. Other 
professionals, teaching assistants, and volunteers taught slightly less than 30% of the 
students. ^ ^ 
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■ In writing, the pattern was somewhat different than in reading and math. Students were 
about as likely to be taught by a teacher other than their own as by “others” [other 
professionals, teacher assistants, or volunteers (44%)]. Students were even less likely 
than in reading and math to be served by their own teacher (6%). 



Figure 16 

Elementary Characteristics of ALP: Hours, Group Size, and Instructors 





Reading 

n=551 


Math 

n=498 


Writing 

n=235 


Attendance 








# Sessions Planned 


42.0 


35.5 


33.5 


# Sessions Attended 


36.1 


30.1 


28.3 


Attendance Rate 


79.9% 


79.5% 


77.4% 


Hours Per Session 


1.6 


1.6 


1.2 


Total Service Hours 


47.7 


39.5 


27.1 


Group Size (Average) 


8.8 


9.1 


9.6 


Instructor: 

Own Teacher (or Own Teacher Plus 


n=63 (12%) 


n=56 (11%) 


n=15 (6%) 


Others) 


n=334 (61%) 


n=299 (60%) 


n=l 16 (49%) 


Another Teacher Only 






Others Only (School Professional, 


n=154 (28%) 


n=142 (29%) 


n=104 (44%) 


Teacher Assistant, Volunteer) 









Multivariate regression models tested whether or not reading and math residuals were affected 
by the amount of service (number of sessions attended times the hours per session), the 
attendance rate, the size of the group, and the instructor. The regression analyses showed that 
none of these variables were significant predictors of the reading or math residual scores. This 
suggests that, within ALP in WCPSS, similar gains were shown for students served by various 
types of instructors, for various hours, and in varying group sizes. It must be kept in mind, 
however, that all students were served in fairly small groups and that the number of sessions 
planned for students often varied depending on need. Since the ALP program was effective 
overall, these analyses provide no evidence that practices need to be changed in these areas. 



Highest-Growth Schools 

Overall, 85% of WCPSS elementary schools showed exemplary growth for Level I and II 
students in 2000-01, up from 80% in 1999-2000. This shows that strong growth was widespread 
for our lowest students, and also that strong exemplary growth will be necessary to accomplish 
the 95% achievement goal. We contrasted the characteristics of the 10 schools with the highest 
growth for Level I and II students (see the figure below) with those of the schools with the 
lowest growth. It is important to recognize that this is primarily a comparison between excellent 
schools and good ones (not poor ones). We compared both characteristics of the student bodies 
as well as the ALP programs based on school profiles, ABC results, and ALP feedback forms. 
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Figure 17 

Ten Highest-Growth WCPSS Elementary Schools for Level I-I3 Students: 
ABC Performance and Exemplar Growth Composites 



Elementary 

School 


Level I/II 


Performance 

Composite 


Exemplary 

Growth 

Composite 


Expected 

Growth 

Composite 


Students in 
Model 


% Free/Reduced- 
Price Lunch 


Leesville Road 


76.50 


2.27 


2.76 


30 


8% 


Morrisville 


77.90 


2.08 


2.59 


32 


3% 


Davis Drive 


77.90 


2.07 


2.61 


27 


4% 


Brooks 


74.30 


1.86 


2.35 


63 


34% 


Green Hope 


77.20 


1.80 


2.32 


10 


3% 


Lockhart 


77.50 


1.78 


2.25 


67 


39% 


Weather stone 


78.10 


1.77 


2.29 


26 


23% 


North Ridge 


77.30 


1.72 


2.21 


44 


30% 


Durant Road 


76.30 


1.70 


2.17 


52 


6% 


Combs 


76.80 


1.66 


2.20 


22 


26% 



On the average, schools with the best gains for Level I and II students were more likely to: 

• have lower concentrations of low income and minority students, 

• provide ALP help outside of the regular school day, 

• use more strategies overall, 

• use frequent assessment to inform instruction, curriculum compacting, and curriculum 
mapping and pacing guides, 

• recognize student learning and staff commitment as strengths of their ALP program, 

• report fewer challenges (low schools mentioned staff recruitment and burnout more 
often), 

• have steady attendance across the school year (with declines more common across the 
year in low schools). 

Both high- and lower-growth schools mentioned student enthusiasm and attendance as strengths 
in their programs. Other similarities were the use of math manipulatives, frequent feedback to 
students, leveled book rooms, and individualized instruction. Both groups tended to do progress 
reports between the regular and ALP teacher (if not the regular teacher) informally, with periodic 
written summaries (with conferences at some schools). Personalized Education Plans (PEPS) 
were mentioned by only two schools as playing a role in this communication. Neither group 
extensively used special electives, extended advisory or team times, or special remediation/ 

enrichment times during the day. Highest-growth elementary schools were larger schools but 
with lower concentrations of Level I-II and low-income students, and they offered more ALP 
hours than lowest-growth elementary schools. 
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More intangible characteristics of schools, such as instructional leadership, staff enthusiasm, 
staff skills and experience, and student expectations also likely play a part in school success. A 
description of approaches in two schools with top gains as of spring 2000 is included in 
Attachment 5. More information on programs in top schools this year can be found in ALP 
Report 1 (E&R Report 01.36) and from the Evaluation and Research Department. 

ALP 6-8 

STUDENT ELIGIBILITY AND PARTICIPATION 

Data sheets returned for individual students indicated that 66.8% of the eligible students actually 
participated in the program. Most of those who did not participate, received other WCPSS 
services. The following figure shows ALP eligibility and participation in 2000-01. 



Figure 18 

Enrollment in ALP 6-8 2000-01 



Grade 


# Students Eligible 
for ALP 


# Students Who 
Participated in 
ALP 


% Students Who 
Participated in 
ALP 


6 


1,515 


1,035 


68.3% 


7 


1,733 


1,174 


67.7% 


8 


1,218 


774 


63.5% 


Total 


4,466 


2,983 


66.8% 



Source: Instructional Assistance Data Sheets 



ATTENDANCE 

As with elementary schools, attendance rates varied greatly across middle schools overall and 
within the various time slots ALP was provided. 

ALP sessions offered during the school day had the highest mean attendance, at 88%. Saturday 
attendance by school ranged from 27% to 1 00%; the mean percentage was 65% attendance. The 
mean attendance percentage for after-school sessions was also 65%, with attendance ranging 
from 10% to 100%. The two year-round schools responding reported intersession attendance of 
77% and 87%. Attendance was up slightly for all time slots compared to 1999-2000 attendance. 
Schools most often reported low attendance (defined as under 60%) for Saturday sessions. 
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Figure 19 

2000-01 Attendance Trends for ALP 6-8 





Saturday 


After School 


During the Day 


Intersession 




(N = 17 of 24) 


(N = 10 of 24) 


(N = 8 of 24) 


(N = 2 of 3) 


Attendance 


% of Schools 


% of Schools 


% of Schools 


% of Schools 


Under 60% 


47.0% 


30.0% 


0% 


0% 


60 - 69% 


11.8% 


20.0% 


0% 


0% 


70 - 79% 


17.7% 


10.0% 


12.5% 


50.0% 


80-89% 


5.9% 


20.0% 


37.5% 


50.0% 


90% and Above 


17.7% 


20.0% 


50.0% 


0% 



Source: ALP Feedback Forms 



Figure 20 

ALP 6-8 Attendance Across Schools: Mean Percentages 





On School Days 


Not on School Days 


During 
the School 
Day 


Before 

School 


After 

School 


Teacher 

Workday 


Saturday 


Inter- 

session 

Days 


2000-01 


88% 


— 


65% 


63% 


65% 


82% 


1999-2000 


N/A 


— 


62% 


60% 


61% 


79% 



Source: ALP Feedback Forms 



Individual Student Attendance 



The attendance of individual students, as determined by the individual data sheets submitted for 
a sample of 1,555 students, was 75% for reading, 71% for math, and 75% for writing. 



Figure 21 

Individual Student Attendance for Instructional Assistance in Middle Schools 



Subject 


# Students 


Mean 

Percentage 


Reading 


1,180 


75% 


Math 


1,070 


71% 


Writing 


373 


75% 
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Students needing help in reading tended to receive the most assistance. Writing help was 
generally provided to students already there for reading or math help (with considerably lower 
enrollment). 

SUCCESSES AND CHALLENGES 

Based on input from the ALP Feedback Form, middle schools considered the greatest success of 
ALP to be staff commitment (80.8%). Student learning and student enthusiasm were also 
mentioned as successes by 53.8% and 42.3% of schools, respectively. 

Schools found the most challenging aspect of the ALP program to be student attendance 
(53.8%). This was less commonly mentioned by elementary school staff (23%). Recruiting 
staff, student motivation, and transportation were the next most frequently cited, with 34.6% of 
schools reporting challenges in these areas. 

IMPACT ON STUDENT ACHIEVEMENT 

Growth for Students Based on ALP Participation 



Level I and II students made strong growth between spring 2000 and spring 2001. Did students 
with needs who were served in ALP show greater growth than students not served in ALP? The 
answer at the middle school level is yes at grades 7 and 8. 

We compared growth for students served in ALP (or ALP and another program) to those not 
served in ALP (most of whom were likely served in another way, eg., exclusively through 
Special Education, ESL, SOAR, SOS, CIS, a private tutor, or parents). Analyses were subject 
specific, including only students who scored in Levels I or II in the specific subject. As an 
external standard, we included ABC expectations for exemplary growth for Level I and II 
students. ABC analyses include students with Level I or II in either reading or math in analysis 
of both subjects. Our analysis was more accurate in tracking improvement for those who truly 
had deficits, but results must be interpreted with more caution because of this difference in 
inclusion rules. 

We checked the characteristics of students participating and not participating in ALP. Those in 
ALP were more likely to be low income (55% vs. 48%) and less likely to be special education 
(39% vs. 61%) than those not involved. We computed mean gains for each group and conducted 
regression analyses, which controlled for pretest scores, free-lunch status, and special education 
status. One caution is that the two groups may have varied in other ways as well. 

As shown in the following figure, middle school students in ALP met or exceeded ABC 
exemplary standards in all three grades in both reading and math; those not served in ALP met or 
exceeded ABC standards in all areas except in grade 6 reading. These overall results are positive 
and point out the benefit of making assistance available to all those in need (which was not 
possible before ALP). 
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ALP also appeared stronger than other support for middle school students except at grade 6. 
Gains in both grades 7 and 8 were stronger for students served in ALP in both reading and math 
(p< .05 except grade 8 reading, which was p<.10). It could be because less assistance is 
available overall at the middle school level. In reading, it could also reflect the fact that reading 
is not provided as a subject on its own at the middle school level. 

Figure 22 

EOG Scale Score Gains Based on ALP Participation 
Relative to ABC Exemplary Standards for Level I-II Students 2000-01 



18 -> 



16 - 



14 - 



■ ABC Exemplary 
HALP 
□ Not ALP 
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Figure 23 

EOG Reading and Math Gains for Students in Level I-II 
Served in ALP 6-8 or Not Served in ALP 



EOG Reading 


Grade 6 


Grade 7 


Grade 8 


# Students 


Gain in SS* 


# Students 


Gain in SS* 


# Students 


Gain in SS* 


ABC Exemplary 


2,405 


3.3 


2,471 


3.5 


2,489 


3.0 


ALP 


639 


3.3 


910 


7.8 


534 


6.2 


Not ALP 


227 


3.1 


355 


6.7 


231 


5.4 


All Level I-II 


866 


3.2 


1,265 


7.5 


765 


5.9 


EOG Math 


Grade 6 


Grade 7 


Grade 8 


# Students 


Gain in SS* 


# Students 


Gain in SS* 


# Students 


Gain in SS* 


ABC Exemplary 


2,405 


6.9 


2,471 


6.2 


2,489 


4.4 


ALP 


527 


8.9 


512 


12.1 


392 


6.9 


Not ALP 


207 


9.2 


254 


10.4 


236 


6.6 


All Level I-II 


734 


9.0 


766 


11.5 


628 


6.8 



Shading Means Group Met ABC Exemplary Growth Standard 

* Gains in SS = Average Scale Score Gain 00 to 0 1 

Note: Most students were served through another WCPSS program. 



FACTORS LEADING TO GREATEST ACHIEVEMENT GAINS 

Schools had a great deal of flexibility in implementing ALP (within broad guidelines), and 
student gains varied considerably across school programs. We therefore examined whether 
student gains differed based on program characteristics. One regression analysis analyzed the 
impact of the time of service on student achievement outcomes, while a second explored the 
impact of type of instructor, hours of service, group size, and instructional approach. 

Timing of ALP Service 



Schools were given more flexibility in 2000-01 to provide ALP services at least partially during 
the day. Roughly 75% of students received ALP help primarily outside of the regular school 
day in 2000-01 while about 15% were served only during the day and 8% served both during the 
day and outside. The regression model tested whether students were more likely to show higher 
standardized residual scores if they were served only outside of the school day, during the school 
day, or both during and outside of the school day. Overall for the middle school analysis, 447 
students were included for reading and 402 for math. As the next chart illustrates, time of 
service did not impact elementary reading residuals, but math residuals were lower if students 
were served outside of the school day only (compared to during the school day only). 
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Figure 24 

Middle School: Importance of Help Outside the School Day 





Time of Day 


N 


Significance 


Reading 


Outside Only 


331 


ns 


During Only 


77 


ns 


Both 


39 


ns 


Math 


Outside Only 


311 


Significantly lower than During Only (p=.013) 


During Only 


59 


Significantly higher than Outside Only (p=.004) 


Both 


32 


ns 



These results suggest that ALP reading programs can be successful regardless of the time of day 
students are served. However, with middle school students receiving ALP help in math, it is 
better to provide at least some service during the day (through electives or other means). For 
students with needs in both reading and math, if students have time for only one elective, math 
may be the better choice (with reading support provided outside of the school day). 

Other Factors Related to ALP Service: Attendance, Group Size, & Instructor 

In addition to examining the timing of ALP, the analysis also considered attendance, group size, 
and instructor. After deleting missing data across variables, 392 middle school students were 
included in the reading analysis, 345 in the math analysis, and 93 in the writing analysis. 
Descriptive statistics for attendance, group size, and instructor are shown in the following figure. 
On average, middle school students planned and attended more ALP sessions in reading (33) 
than in math (24) and writing (3 1). Attendance rates were comparable in reading and math 
(65%) but higher in writing (77%). Hours per session (1.8) were identical in reading and math 
but lower in writing (1 .4). Total service, as derived by multiplying the number of sessions 
attended by the hours per session, was highest in reading, with similar hours provided in math 
and writing. (We actually expected to see less service in ALP in writing than in reading or math, 
since ALP funding is based on reading and math, writing is not part of the system’s achievement 
goal, and writing is tested less often by the state. However, new promotion standards include 
writing performance as one criterion, which may contribute to these results.) 

Group size averaged about 9 students in both reading (9.5) and math (9.4) but groups were larger 
in writing (1 1 .3). The standard deviation for group size was about 2.8 students in reading and 
math, and much larger in writing (5.5 students) 

The instructor variable consisted of three categories for ALP, reflecting service by. a) the 
student’s teacher (often plus others), b) another teacher only, or c) other professionals, teaching 
assistants, and volunteers. ALP students were most often served by teachers other than their 
regular teacher for the subject in reading, math, and writing. 
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their own. Other professionals, teaching assistants, and volunteers taught slightly less 
than 1 5% of the students. 

■ The pattern for writing was somewhat different from reading and math, with the type of 
instructor was more evenly distributed. While teachers were still most likely to be 
served by teachers other than their own (44%), students were also more likely to be 
served by their own teacher (31%) or by other professionals, teacher assistants, or 
volunteers (25%). 



Figure 25 



Middle School 


Characteristics of ALP Time and Instructors 




Reading 


Math 


Writing 




n=392 


n=345 


n=93 


Attendance 








# Sessions Planned 


42.5 


32.9 


37.5 


# Sessions Attended 


33.3 


24.3 


30.9 


Attendance Rate 


66.3% 


65.3% 


77.3% 


Hours Per Session 


1.8 


1.8 


1.4 


Total Service Hours 


40.7 


30.1 


30.5 


Group Size (Average) 


9.5 


9.4 


11.3 


Instructor: 








Own Teacher (or Own 


n=50 (13%) 


n=45 (13%) 


n=29 (31%) 


Teacher Plus Others) 








Other Teacher Only 


n=288 (73%) 


n=249 (72%) 


n=41 (44%) 


Others Only (School 


n=54 (14%) 


n=51 (15%) 


n=23 (25%) 


Professional, Teacher 








Assistant, Volunteer) 









Multivariate regression models tested whether or not reading and math residuals were affected 
by the amount of service (number of sessions attended times the hours per session), the 
attendance rate, the size of the group, and the instructor. The regression analyses showed that 
none of these variables was significant predictors of the reading or math residual scores. These 
results suggest that, within ALP in WCPSS, similar gains were shown for students served by 
various types of instructors, for various hours, and in varying group sizes. It must be kept in 
mind, however, that all students were served in fairly small groups, and that the number of 
sessions planned for students often varied depending on need. Since the ALP program was 
effective overall, these analyses provide no evidence that practices need to be changed in these 
areas. 

Highest-Growth Schools 

Among the five highest-growth middle schools were two of WCPSS’s three year-round schools. 
The top schools offered an average of 59.8 hours of ALP instruction, compared to 91 hours for 
the five lowest-growth schools. Four of the five high-growth schools offered mixed session 
types for ALP, including Saturday hours. Session attendance ranged from 50.0% (before/after 
school) to 100% (during the school day). All of the high-growth schools utilized a wide variety 
of supplemental materials, with most specifically mentioning manipulatives. The five high- 
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growth schools had an average of 136 Level I and II students in their models; low-growth 
schools averaged 142.8 Level I and II students. 

High-growth schools were more likely to: 

• Begin their ALP programs early in the school year. 

• Offer more than one session time for ALP assistance. 

• Note high levels of parent cooperation with the ALP program. 

• Report that the new WCPSS promotion/retention policy increased parent and student 
willingness to participate in ALP. 

• Utilize a higher number of strategic approaches to support student success (e.g., teaming 
across grades for instruction, curriculum mapping, and extended advisory or team time). 

• Used a greater range of supplemental materials such as software. 

• Report student enthusiasm as one of the program’s successes. 

• Find that staff recruitment was not a notable challenge. 

Figure 26 

Five Highest-Growth WCPSS Middle Schools: 

ABC Performance and Exemplary Growth Composites 



School Name 


Level I/H 


Performance 

Composite 


Expected 

Growth 

Composite 


Exemplary 

Growth 

Composite 


Students in 
Model 


Durant Road (YR) 


72.1 


“ 1.54 


1.25 


162 


Leesville Road 


69.0 


1.43 


1.14 


107 


West Lake (YR) 


70.6 


1.30 


1.02 


126 


Daniels 


67.3 


1.18 


0.90 


138 


East Cary 


67.7 


1.01 


0.73 


147 



Middle schools with highest and lowest growth had approximately the same number of students 
participating in their ALP programs (136 students for highest-growth schools and 142.8 for 
lowest-growth schools), and comparable percentages of eligible students participating (68.6% for 
the highest and 65.8% for the lowest). Both highest- and lowest-growth schools used math 
manipulatives, high-interest reading materials, and learning games. 

EDUCATIONAL SIGNIFICANCE 

As school systems nationwide struggle to determine how to meet the new challenges of ESEA, 
the experience of WCPSS over the last several years may provide some useful tips and lessons. 
The combination of the state ABC model (with performance and growth components) and the 
95% achievement goal, have combined to create a balance and tension between helping all 
students grow (necessary to reach growth standards) and providing extra help to those below 
grade level in performance (the 95% goal). The new ESEA model accomplishes this in a 
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different way, with achievement performance targets for all subgroups, but the end goal is 
similar, albeit even tougher (95% or 100% of students performing on grade level). Some general 
lessons from our experience in WCPSS are that: 

• Extended learning opportunities during the year can make a positive difference for low 
achievers. Other research conducted in WCPSS indicates assistance during the year was 
more helpful than summer school (Holdzkom, 2002). Success will require additional 
resources and a concerted effort over time. 

• The 100% standard set by ESEA will be nearly impossible to meet if only due to the 
measurement error inherent to all tests (95% is tough enough). States who do not already 
have state systems may set lower standards for “grade level” than states like North Carolina 
who have had tests in place. 

This study also relates to the value of our work in some important ways. 

• Despite the lack of random assignment, we were able to provide the school system with 
some useful data about program effectiveness. (Encouraging school systems to pilot 
programs in more experimental ways is still a good goal, however, especially with new 
ESEA guidelines about using practices proven effective through research.) 

• As evaluators, we must go beyond system-wide analyses based on regression if we are to 
have an impact on school practice. Disaggregated data, in this case our comparisons of 
most effective and least effective school practices, have influenced school practice in 
WCPSS more than the system trends. This, of course, was enhanced through the addition 
of some qualitative analysis. A second example is summarizing the number of students 
moving down, as well as up, to grade level standards. This has influenced schools to fit 
slightly higher scoring students into ALP as room allowed, and may influence the way 
eligibility is determined as well. Often descriptive analyses are as valuable to practioners 
as more sophisticated statistical analyses. Mixed method designs may yield the best blend 
to encourage use of findings. 
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